Silicon Valley players are poised to benefit. One of them is Palmer Luckey, the founder of the virtual-reality headset company Oculus, which he sold to Facebook for 2 billion. After Luckey's highly public ousting from Meta, he founded Anduril, which focuses on drones, cruise missiles, and other AI-enhanced technologies for the US Department of Defense. The company is now valued at 14 billion. My colleague James O'Donnell interviewed Luckey about his new pet project: headsets for the military.

In this paper, we introduce a novel system designed to enhance customer service in the financial and retail sectors through a context-aware 3D virtual agent, utilizing Mixed Reality (MR) and Vision Language Models (VLMs). Our approach focuses on enabling data-driven and empathetic interactions that ensure customer satisfaction by introducing situational awareness of the physical location, personalized interactions based on customer profiles, and rigorous privacy and security standards. We discuss our design considerations critical for deployment in real-world customer service environments, addressing challenges in user data management and sensitive information handling. We also outline the system architecture and key features unique to banking and retail environments. Our work demonstrates the potential of integrating MR and VLMs in service industries, offering practical insights in customer service delivery while maintaining high standards of security and personalization.

We introduce iTeach, a Mixed Reality (MR) framework to improve robot perception through real-time interactive teaching. By allowing human instructors to dynamically label robot RGB data, iTeach improves both the accuracy and adaptability of robot perception to new scenarios. The framework supports on-the-fly data collection and labeling, enhancing model performance, and generalization. Applied to door and handle detection for household tasks, iTeach integrates a HoloLens app with an interactive YOLO model. Furthermore, we introduce the IRVLUTD DoorHandle dataset. DH-YOLO, our efficient detection model, significantly enhances the accuracy and efficiency of door and handle detection, highlighting the potential of MR to make robotic systems more capable and adaptive in real-world environments. The project page is available at https://irvlutd.github.io/iTeach.

We create an innovative mixed reality-first social recommendation model, utilizing features uniquely collected through mixed reality (MR) systems to promote social interaction, such as gaze recognition, proximity, noise level, congestion level, and conversational intensity. We further extend these models to include right-time features to deliver timely notifications. We measure performance metrics across various models by creating a new intersection of user features, MR features, and right-time features. We create four model types trained on different combinations of the feature classes, where we compare the baseline model trained on the class of user features against the models trained on MR features, right-time features, and a combination of all of the feature classes. Due to limitations in data collection and cost, we observe performance degradation in the right-time, mixed reality, and combination models. Despite these challenges, we introduce optimizations to improve accuracy across all models by over 14 percentage points, where the best performing model achieved 24% greater accuracy.

Because we in Silicon Valley are newness junkies, it can feel like an act of sabotage to have memories, but, for better or worse, I have them. It's been more than forty years since I co-founded the first company to make headsets and software for simulated experiences, and came up with familiar terms like virtual and mixed reality. Since then, virtual reality has flooded the public imagination in waves; back in the nineteen-eighties, for instance, it had quite a presence in movies, cartoons, TV shows, the occasional arcade game, and a few early consumer products, like the Nintendo Power Glove. I still love V.R. But, these days, I sense that what I experience of it, what I enjoy in it, is different from what it has come to mean to many enthusiasts. Back then, at the beginning, did I talk about V.R. like the people I disagree with now? I did occasionally promote V.R. as an alternate cosmos that might swallow us all to good effect. I don't agree with that sort of talk now, but at the time the joy of being edgy and extreme was too great to resist.

Apple hopes the Vision Pro will fundamentally change how we interact with our devices--that once freed from the constraints of a smartphone or tablet screen, we'll embrace "spatial computing," as the glitzy promo video shows. Gesture and eye tracking identifies where your focus is, allowing you to interact with apps without pressing buttons or a screen. That could be great for consumers. Apple explained that existing apps designed for the iPad will work on visionOS, the operating system powering the Vision Pro, without any changes. But those iPad apps will be displayed within a metaphorical window, losing much of the functionality provided by mixed reality.

That was the first thing I heard from one excited WWDC attendee as I waited to test Apple's Vision Pro mixed reality headset. That level is excitement is exactly what Apple is hoping for. Realistically, not everyone will be able to afford a $3,499 device. But if Apple can get mainstream consumers excited about the idea of spatial computing, then it'll be able to make a bigger splash when it inevitably unveils a more affordable follow-up. After spending thirty minutes with the Vision Pro, my reaction is more tempered than that excitable attendee.

A novel method to control the Spot robot of Boston Dynamics by Hololens 2 is proposed. This method is mainly designed for people with physical disabilities, users can control the robot's movement and robot arm without using their hands. The eye gaze tracking and head motion tracking technologies of Hololens 2 are utilized for sending control commands. The movement of the robot would follow the eye gaze and the robot arm would mimic the pose of the user's head. Through our experiment, our method is comparable with the traditional control method by joystick in both time efficiency and user experience. Demo can be found on our project webpage: https://zhangganlin.github.io/Holo-Spot-Page/index.html

Apple's big developer conference kicks off June 5th, and all the signs point to the company's mixed reality headset making its first appearance. The tech giant has been acquiring headset-friendly startups for years, and if the rumors are true, Apple's stand-alone device (possibly called Reality Pro) may be more powerful than the Meta Quest Pro and many other high-end headsets. It could pack 4K resolution per eye, with full body-motion tracking. It may require an external battery pack and last for just two hours on a charge, but it would be relatively light and slim. It will also likely land with a new platform (maybe called xrOS) designed with mixed reality in mind.

Working in the UX industry means living in a constant state of flux. Every day seems to bring new technologies, skills, business challenges, and user expectations to absorb. That's why every December, I eagerly await the release of UX Collective's State of UX report, in which authors Fabricio Teixeira and Caio Braga (plus collaborators) synthesize thousands of articles in order to put the past year into perspective and highlight emerging themes for the road ahead. The underlying theme for this year's report was anxiety. Massive layoffs at tech stalwarts like Facebook, Google, and Amazon, along with headlines about an economic slowdown, have some designers thinking about how to recession-proof their jobs. With headcounts shrinking, design teams are expected to do more with less, and former managers are returning to hands-on work.